Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up filtered disjunctions by loading the filter into a bit set. #14024

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Speed up filtered disjunctions by loading the filter into a bit set. #14024

wants to merge 4 commits into from
+115 −52

Conversation

jpountz
Copy link
Contributor

@jpountz jpountz commented Nov 27, 2024

This helps skip the overhead of reordering the heap on every doc, so the benefit is especially visible with many clauses.

@jpountz
Speed up filtered disjunctions by loading the filter into a bit set.
a96a152 
This helps skip the overhead of reordering the heap on every doc, so the
benefit is especially visible with many clauses.
@jpountz jpountz added this to the 10.1.0 milestone Nov 27, 2024
@jpountz
Copy link
Contributor Author

jpountz commented Nov 27, 2024

All filtered disjunctions but FilteredOr2Terms2StopWords get a speedup:

                            TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
      FilteredOr2Terms2StopWords      151.97      (0.9%)      144.44      (0.8%)   -5.0% (  -6% -   -3%) 0.000
                     OrStopWords       33.54      (8.5%)       33.10      (9.5%)   -1.3% ( -17% -   18%) 0.716
                        Or3Terms      174.11      (5.0%)      172.37      (5.9%)   -1.0% ( -11% -   10%) 0.641
                 DismaxOrHighMed       87.67      (4.6%)       86.85      (3.4%)   -0.9% (  -8% -    7%) 0.556
              Or2Terms2StopWords      163.99      (5.0%)      162.75      (5.8%)   -0.8% ( -10% -   10%) 0.720
                  FilteredPhrase       26.21      (2.5%)       26.07      (3.1%)   -0.5% (  -5% -    5%) 0.633
                    AndStopWords       30.02      (6.2%)       29.86      (5.6%)   -0.5% ( -11% -   12%) 0.826
                      OrHighHigh       53.85      (1.2%)       53.62      (2.1%)   -0.4% (  -3% -    2%) 0.531
                    CombinedTerm       32.39      (3.0%)       32.27      (4.4%)   -0.4% (  -7% -    7%) 0.805
                       OrHighMed      200.12      (1.1%)      199.54      (1.8%)   -0.3% (  -3% -    2%) 0.626
                 AndHighOrMedMed       46.77      (0.9%)       46.63      (2.1%)   -0.3% (  -3% -    2%) 0.650
                 CountOrHighHigh       76.39      (0.6%)       76.17      (0.7%)   -0.3% (  -1% -    1%) 0.285
             FilteredAndHighHigh       63.19      (1.5%)       63.05      (1.6%)   -0.2% (  -3% -    2%) 0.716
                 CountAndHighMed      170.83      (1.4%)      170.46      (1.4%)   -0.2% (  -3% -    2%) 0.701
                DismaxOrHighHigh       70.86      (4.4%)       70.75      (3.1%)   -0.1% (  -7% -    7%) 0.922
                       CountTerm     9302.53      (3.7%)     9291.97      (4.0%)   -0.1% (  -7% -    7%) 0.940
                  CountOrHighMed      144.44      (1.1%)      144.29      (1.0%)   -0.1% (  -2% -    2%) 0.795
              FilteredAndHighMed      127.09      (2.8%)      126.96      (2.6%)   -0.1% (  -5% -    5%) 0.926
            FilteredAndStopWords       49.12      (1.7%)       49.08      (2.0%)   -0.1% (  -3% -    3%) 0.903
                       And3Terms      170.32      (4.0%)      170.20      (4.0%)   -0.1% (  -7% -    8%) 0.963
                CountAndHighHigh       58.02      (1.0%)       57.98      (1.3%)   -0.1% (  -2% -    2%) 0.904
                      DismaxTerm      648.52      (3.3%)      648.23      (3.4%)   -0.0% (  -6% -    6%) 0.973
                AndMedOrHighHigh       59.68      (0.7%)       59.72      (1.5%)    0.1% (  -2% -    2%) 0.892
                          OrMany       19.73      (5.7%)       19.74      (5.4%)    0.1% ( -10% -   11%) 0.976
             And2Terms2StopWords      160.42      (4.0%)      160.73      (3.5%)    0.2% (  -6% -    7%) 0.895
               FilteredAnd3Terms      193.11      (1.7%)      193.54      (1.6%)    0.2% (  -3% -    3%) 0.739
     FilteredAnd2Terms2StopWords      198.70      (1.4%)      199.50      (1.4%)    0.4% (  -2% -    3%) 0.454
                     AndHighHigh       42.13      (1.4%)       42.31      (1.2%)    0.4% (  -2% -    3%) 0.413
                      AndHighMed      123.75      (1.0%)      124.35      (1.3%)    0.5% (  -1% -    2%) 0.279
              CombinedAndHighMed       56.43      (1.6%)       56.77      (2.1%)    0.6% (  -3% -    4%) 0.412
             CombinedAndHighHigh       15.45      (1.5%)       15.54      (1.9%)    0.6% (  -2% -    4%) 0.348
                      OrHighRare      281.45      (6.0%)      283.37      (6.6%)    0.7% ( -11% -   14%) 0.782
                     CountPhrase        4.49      (1.5%)        4.52      (1.1%)    0.7% (  -1% -    3%) 0.172
                        PKLookup      281.29      (1.6%)      284.30      (1.5%)    1.1% (  -2% -    4%) 0.082
                    FilteredTerm      160.14      (2.8%)      161.98      (2.1%)    1.1% (  -3% -    6%) 0.238
                FilteredOr3Terms      170.41      (0.8%)      172.42      (1.3%)    1.2% (   0% -    3%) 0.007
              CombinedOrHighHigh       18.28      (3.8%)       18.56      (4.8%)    1.6% (  -6% -   10%) 0.358
               CombinedOrHighMed       69.85      (4.1%)       70.95      (5.2%)    1.6% (  -7% -   11%) 0.388
             FilteredOrStopWords       44.83      (1.8%)       47.08      (2.9%)    5.0% (   0% -    9%) 0.000
              FilteredOrHighHigh       66.63      (1.2%)       70.79      (1.9%)    6.2% (   3% -    9%) 0.000
               FilteredOrHighMed      158.80      (0.8%)      170.21      (1.2%)    7.2% (   5% -    9%) 0.000
                  FilteredOrMany       16.89      (4.0%)       25.68      (1.1%)   52.1% (  45% -   59%) 0.000

@javanna javanna modified the milestones: 10.1.0, 10.2.0 Dec 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
10.2.0
Development

Successfully merging this pull request may close these issues.
2 participants
@jpountz @javanna